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Figure 1: Nodes and physical connectivity 
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Figure 2a) JOIN Protocol: PROCLAIM message 
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Figure 2b) JOIN Protocol: JOIN message 
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Figure 2e) JOIN Protocol: C0MM1T_BCAST message 
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Figure 2f) JOIN Protocol: COMMIT and COMMIT_BCAST_ACK messages 




Figure 2g) JOIN Protocol: new group formed after completion of protocol 
Figure 2: JOIN protocol 
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Figure 3a) DEATH Protocol: initial state: heartbeat ring 
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Figure 3b) DEATH Protocol: DEATH message 




Figure 3c) DEATH Protocol: PTC message 
Figure 3: DEATH protocol 
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NODE.CONNECTIVrrY 
Figure 4a) Node Reachability Protocol: NODE_CONNECTIVITY message 
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Figure 4c) Node Reachability Protocol: forwarding of GROUP_CONNECTIVITY message 



Figure 4: Node reachability protocol: NODE_CONNECnVITY and GROUP_CONNECTIVITY 
messages 
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5a) Initial situation 



Figure 5: Topology Propagation Scenario: node death 
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5b) Node 2 dies: Nodes 1, 3, and 4 form AMG A_2 
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5c) GCM for AMG A_2 is propagated to all nodes 
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Figure $) Inconsistency caused by quick daemon restart in the presence of different detection times 
for each network: the daemon on node 1 goes down and is restarted, but this is never detected 
by node 2. 
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Figure f ) Inconsistency caused by temporary communication problem in an adapter: node 1 
is not the GL or CP in its group. Node 1 never notices other nodes as down, though the others 
do see node 1 being unreachable. 
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Figure Inconsistency caused by temporary communication problem in an adapter: node 1 
is the GL or CP in its group. Node 1 never sees the other nodes as unreachable, while the 
others do see node 1 as unreachable for a period. 
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Figure 1&) Adapter IDs and Group IDs. An adapter ID has the IP address 
of the adapter and and instance number. The Group ID has the IP address 
of the Group Leader and an instance number that changes each time the 
group changes. 
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Figure Format of the protocol packets that are sent over the network 
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Figure !&) Adapter and Group IDs when the daemon at node 1 terminates and is 
restarted. 
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Figure A "live" node detects that a remote daemon restarted.The Group ID of the 
message is different from node 2's, while the address of the sender is listed on node 2's 
group membership. 
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Figure Ma) A daemon that is restarted detects that a previous instance used to belong to 
an AMG because of heartbeat messages that it receives while in a singleton group. 
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Figure Wo) Continuation 
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Figure ®a) Solution to the Quick Communication Interruption Problem, initial state: 
nodes 1,2, and3 are part of the same AMG,. Node 3's adapter suffers a temporary 
failure. 
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Figure 9$b) Solution to the Quick Communication Interruption Problem. Node 3's 
adapter suffers a temporary failure. Node 2 commits a new AMG, while node 3 is still 
in the process of missing HBs from its neighbor 
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Figure $£?c) Solution to the Quick Communication Interruption Problem. Node 3 sends 
a PTC when it stops receiving HBs from its upstream neighbor. The PTCs are rejected 
because of the discrepancy in the last_stable_group results. 
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Figure ^d) Solution to the Quick Communication Interruption Problem. Since node 3 
does not get replies to its PTC messages, it is forced to form a singleton group. At this 
point, it updates last_stable_group. From then on node 3's PTC are accepted again. 



Nodel 



1.1.1.1 



7228 



1.1.1.3 



7820 



3® 



Node 3 (GL) 



Adapter ID 



Group ID 



1.1.1.3 



7687 



1.1.1.3 



7820 



1.1.1.3 


1.1.1.2 


1.1.1.1 


Group (AMG) 


1.1.1.3 


1.1.1.2 


1.1.1.1 


7687 


7259 


7228 


7687 


7259 


7228 
















1.1.1.3 


1.1.1.2 


1.1.1.1 


last stable_group 


1.1.1.3 


1.1.1.2 


1.1.1.1 


7687 


7259 


7228 


7687 


7259 


7228 



Communication glitch 
in node l's adapter. 



Figure ifea) Solution to the Quick Communication Interruption Problem, initial state: 
nodes 1,2, and3 are part of the same AMG,. Node l's adapter suffers a temporary 
failure. 



Nodel 



1.1.1.1 



7228 



1.1.1.3 



7820 



^p/ru f£d#£ d 0 J 6 US/ 



Adapter ID 



Group ID 



Node 3 



1.1.1.3 



7687 



1.1.1.3 



7884 



1.1.1.3 


1.1.1.2 


1.1.1.1 


Group (AMG) 


1.1.1.3 


1.1.1.2 


7687 


7259 


7228 


7687 


7259 














1.1.1.3 


1.1.1.2 


1.1.1.1 


last stable_group 


1.1.1.3 


1.1.1.2 


7687 


7259 


7228 


7687 


7259 



Figure lib) Solution to the Quick Communication Interruption Problem, Node 1 's 
adapter suffers a temporary failure. Node 3 commits a new AMG, while node 1 is still 
in the process of missing HBs from its neighbor 
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Figure jKfe) Solution to the Quick Communication Interruption Problem. Node 1 
dissolves its group and forms a singleton unstable group. Note that because the 
group is unstable, there is no change in last_stable_group. 
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Figure d»d) Solution to the Quick Communication Interruption Problem. Node 3 sends 
a PTC when node 1 responds the PROCLAIM message with a JOIN. The PTCs are rejected 
because of the discrepancy in the last_stable_group results. 
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Figure We) Solution to the Quick Communication Interruption Problem. Since node 3 
does not get replies to its PTC messages, it is eventually forced to form a singleton group. 
At this point, it updates last_stable_group. From then on node 3's PTC are accepted again. 



